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Summary 


This  paper  puts  forward  a  new  paradigm  for  thinking  about  how  better  ways 
for  assessing  candidates  and  for  training  employees  increase  efficiency.  It 
transcends  the  limitations  of  the  two  currently  most  popular  paradigms  for  doing 
so:  the  psychologists*  “utility”  models  and  economists’  decentralized  matching 
models.  In  contrast  to  these  models,  this  new  paradigm  identifies  and  includes 
three  distinct  ways  by  which  improved  assessment  methods  increase  efficiency.  It 
also  points  the  way  towards  a  method  for  quantifying  all  the  benefits  of  using 
improved  methods  of  assessment  candidates.  Moreover,  this  new  paradigm  can  deal 
with  the  benefits  of  improved  training  methods  as  well  as  those  of  improved 
assessment  methods.  So,  it  provides  a  contest  in  which  an  analyst  can  trade-off 
expenditures  for  improving  assessment  methods  against  expenditures  for 
improving  training  methods. 

This  paradigm  builds  on  a  fifty-year  old  tradition  in  applying  mathematical 
programming  techniques  to  improve  job  assignment  processes  in  the  military 
services.  It  incorporates  advances  both  in  representing  quantitatively  on-the-job 
performance  and  in  stochastic  programming  and  integer  techniques. 

The  paper  begins  with  the  concept  of  an  applicant’s  expected  first-  term 
productivity.  Then,  it  lays  out  the  structure  of  the  mathematical  programming 
model.  Next,  the  paper  explains  how  improvements  in  assessment  methods  and 
training  programs  can  be  linked  to  the  model.  Finally,  it  shows  how  the  duel  of  the 
programming  model  and  its  shadow  prices  can  be  used  to  estimate  the  beuefits  of 
the  improvemeut. 
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The  Beneflts  from  Better  Testing  and  Training: 
A  Comprehensive  Framework 


I.  Introduction 


A  traditional  example  ...of  comparative  advantage  is  the  case 
of  the  best  lawyer  in  town  who  is  also  the  best  typist  in  town. 
Will  she  not  specialize  in  law  and  leave  typing  to  a  secretary? 
How  can  she  afford  to  give  up  precious  time  from  the  legal 
field,  where  her  comparative  advantage  is  very  great,  to 
perform  typing  activities  in  which  she  is  efficient  but  in  which 
she  lacks  comparative  advantage?  Or  look  at  it  from  the 
secretary’s  point  of  view.  She  is  less  efficient  than  the  lawyer 
in  both  activities;  but  her  relative  disadvantage  compared  with 
the  lawyer’s  is  least  in  typing.  Relatively  speaking,  the 
secretary  has  a  comparative  advantage  in  typing  (Samuelson, 
p.  669) 


Better  methods  of  evaluating  applicants  and  predicting  their  perforniance  in 
Air  Force  jobs  can  increase  the  efficiency  with  which  the  Air  Force  accomplishes  it 
mission  in  three  distinct  ways. 

1.  Selecting  More  Productive  Applicants.  Better  methods  increase  the  Air 
Forces’  chances  of  selecting  more  productive  individuals  from  its 
applicant  pool. 

2.  Capitalizing  on  Individuals’  Relative  Strengths.  As  the  opening 
quotation  froln  Samuelson’s  Economics  suggests,  the  Air  Force  can 
increase  its  efficiency  by  assigning  selected  individuals  to  jobs  in  which 
they  have  comparative  advantages. 

3.  Cutting  Staffing  Safety  Margins.  Some  recruits  will  fail  to  perform  as 
well  as  expected.  So,  to  ensure  that  its  can  carry  out  its  missions,  the  Air 
Force  has  to  provide  for  staffing  safety  margins.  Predicting  individuals’ 
job  performance  for  accurately  means  that  the  staffing  safety  margins 
required  to  ensure  mission  accomplishment  could  be  cut. 

Industrial  psychologists  have  addressed  the  benefits  of  testing  within  their 
“utility  model”  framework  (Cascio,  1987,  pp.  270-296;  Roach.  1984).  Their 
framework  assumes  one  or  more  openings  for  a  single  kind  of  job.  It  deals  with  the 
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interrelationships  between  five  variables:  (1)  the  ratio  of  openings  to  applicants,  (2) 
the  job  performance  measure,  (3)  the  correlation  of  the  selection  score  and  job 
performance,  (4)  the  cutoff  test  score,  and  (5)  the  cost  of  testing.  • 

The  utility  model  framework  deals  only  with  the  first  of  the  three  ways  of 
increasing  efficiency — selecting  more  productive  applications.  Since  its  assumes  a 
single  kind  of  job,  it  cannot  deal  with  the  efficiency  gains  available  through  more 
fully  exploiting  selected  individuals’  comparative  advantages.  Moreover,  the 
framework  has  not  been  used  to  address  the  required  size  of  the  staffing  safety 
margins. 

In  the  1960’s,  economists  began  to  develop  another  framework  for  studying 
assignment  problems  (Gale  &  Shapley,  1992).  Dale  Mortensen  (1988)  surveyed 
their  approach  and  contributions.  Their  framework,  which  has  been  applied  to 
both  the  job  market  and  the  marriage  market  (Becker,  1973),  assumes  a  large 
number  of  agents  who  are  searching  for  employees  and  jobs  or  husbands  and  wives. 

Within  this  framework,  economists  have  focused  on  identifying  the 
conditions  under  which  decentralized  search  processes  would  result  in  an  optimal 
set  of  matches  and  on  the  incident  of  search  and  information  production  costs 
(MacDonald,  1980).  Most  the  specific  models  within  this  general  framework 
reflect  extreme  assumptions  concerning  what  is  known  abont  job  applicants  and 
potential  mates;  these  models  assume  either  job  performance  and  martial  happiness 
can  be  predicted  perfectly  and  costlessly  when  a  contact  is  first  made  or  they  cannot 
be  known  nntil  the  persons  hired  or  married.  Both  its  focus  of  decentralized 
decision  making  and  its  extreme  assumption  about  what  is  known  about  applicants 
and  potential  mates  limits  its  usefulness  as  a  bases  for  estimating  the  benefits  to  the 
Air  Force  of  better  ways  for  evaluating  applicants. 

This  paper  puts  forward  a  new  framework  for  understanding  the  benefits 
provided  by  better  ways  for  evaluating  applicants.  This  new  framework  transcends 
the  limitations  of  both  psychologists’  utility  model  and  economists’  decentralized 
matching  models.  It  encompasses  all  three  ways  by  which  better  assessment 
methods  can  increase  the  Air  Force’s  efficiency  and  points  towards  a  method  for 
qnantifying  the  comprehensive  benefits  of  better  assessment  methods.  In  addition, 
the  new  framework  is  general  enough  so  that  one  can  deal  with  the  benefits  of  better 
training  methods  as  well  as  those  of  better  assessment  methods.  So,  it  provides  an 
analytic  paradigm  in  which  an  analyst  can  compare  the  benefits  and  costs  of  better 
assessment  methods  with  those  of  better  training  methods. 

This  framework  extends  a  fifty-year  old  tradition  in  applying  mathematical 
programming  techniques  to  improve  job  assignment  processes  in  the  military 
services  (D.  F.  Votaw  &  J.  T.  Daily,  1952;  J.  H.  Ward,  Jr.,  1959;  R.L.  Fernandez  & 

J.  B.  Garfinkle,  1985;  R.  L.  Fernandez,  1987;  M.  Pina,  M.  S.  Emerson,  D.  L. 
Leighton,  &  W.  Cummings,  1988).  Its  used  both  new  ways  of  representing 
quantitatively  on-the-job  performance  and  stochastic  programming  techniques  to 
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deal  with  improvements  in  assessment  and  training  methods  in  a  mathematical 
programming  structure. 

The  next  sections  develop  the  analysis.  First,  the  concept  of  an  applicant’s 
expected  first-  term  productivity  is  described  and  explained.  Then,  the  structure  of 
the  mathematical  programming  model  is  laid  out.  The  paper  then  turns  to  how 
improvements  in  assessment  methods  and  training  programs  could  be  linked  to  the 
model.  Next,  the  use  of  the  duel  of  the  programming  model  and  its  shadow  prices  to 
estimate  the  benefits  of  the  improvement  are  explained.  Finally,  the  implications  of 
the  new  framework  for  assessing  the  benefits  of  improving  Air  Force  selection, 
classification,  and  training  processes  are  discussed. 
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n.  Expected  First-Term  Productivity. 

Once  a  recruit  is  trained  and  given  his  first  assignment,  his  productivity  will 
increase  as  he  acquires  experience.  For  an  Air  Force  Specialty  Code  (AFSC),  an 
applicant’s  expected  total  first-term  producity  can  be  represented  by  the  area  under 
the  curve  in  Figure  1. 


Figure  1 

Expected  First-Term  Productivity 
of  Annlicant  i  in  AFSC  i 


Expected 

Productivity 


Months  Enlisted 

In  Figure  1,  applicant  j’s  productivity  is  AFSC  i  is  expressed  at  a  specific  point  in 
this  first-term  career — ^y(t)ij — as  a  fraction  of  some  standard.  The  standard  used  in 
this  study  is  the  average  productivity  of  airmen  in  AFSC  i  with  journeyman  skills 
and  with  four  years  of  service.  Other  benchmarks  are  possible  and  could  have  been 
chosen  without  altering  the  logic  of  this  study. 

As  shown  in  Figure  1,  an  initial  section  of  applicant  j’s  first-term 
productivity  curve  might  lie  below  the  horizontal  axis.  Such  an  initial  below-the- 
axis  segment  implies  negative  productivity  during  the  initial  months  of  applicant  j’s 
first  job  assignment  in  AFSC  i.  Negative  productivity  would  occur  if  other  members 
of  his  first  unit  had  to  reduce  their  contributions  to  unit  output  to  train  him  and  if 
their  reductions  exceeded  his  initial  contribution  to  his  unit’s  output. 

The  productivity  curve  concept  has  strong  empirical  support.  Haggstrom, 
Chow,  and  Gay  (1984)  fitted  curves  for  48  enlisted  occupations  in  the  Army,  Navy, 
and  Air  Force  using  data  from  almost  30,000  supervisor  surveys;  the  surveys  ask 
supervisors  to  provide  their  subjective  evaluations  of  productivity  contributions  of 
specific  individuals.  Carpenter  and  Monaco  (1987,  pp.  32-45)derived  a  productivity 
curve  for  the  Avionics  Communications  Maintenance  Specialty  based  on 
supervisors’  assessments  of  the  times  that  airmen  required  to  perform 
representative  clusters  of  tasks.  Fernandez  (1987,  pp.  23-25)  refers  to  data 
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collectSed  by  the  RAND  Corp.  on  the  actual  times  required  to  complete  specific 
tasks  in  the  Aerospace  Ground  Equipment  Specialty;  it  also  supports  the  hypotheses 
of  a  rising  productivity  curve. 

The  height  of  the  first-term  productivity  curve  has  been  related  to  measures 
of  ability.  Marcus  and  Quester  (1984)  demonstrated  a  positive  relationship  for 
Navy  using  subjective  evaluation  data  from  surveys.  Carpenter  and  Monaco  (1987) 
demonstrated  a  positive  overaU  relationship  for  the  Avionics  Communications 
Maintenance  Specialty  using  subjective  estimates  of  completion  times  for  task 
clusters, 

Total  first-term  productivity  is  defined  as  the  area  between  the  productivity  curve 
and  the  horizontal  axis.  In  Haggstrom,  Chow,  and  Gay’s  results  (1984,  pp.  72-93), 
first-term  productivity  averaged  1.9q  equivalent  man-years  of  five  “high-skiir  Air 
Force  Specialties  (AFSs),  2.34  equivalent  man-years  for  eleven  “medium  skill” 

AFSs,  and  2.56  equivalent  man-years  for  six  “low  skill”  AFSs. 

Many  selectees  will  not  remain  in  the  Air  Force  for  all  four  years  of  their 
first  terms .  Define  r(t)ij  as  the  likelihood  that  applicant  j  will  be  on  active  duty  in 
month  t  if  he  is  assigned  to  AFSC  i.  Then  the  expected  first  term  productivity  of 
applicant  j  if  he  would  be  assigned  to  AFSC  i  ~  mij— is  defined  by  the  integral: 

m  ij  =  ^]r{t)ijy{t) adt  (l ) 

0 
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III.  An  Analytic  Framework 


This  section  outlines  a  framework  for  estimating  the  benefits  of  better 
assessment  methods  and  training  programs.  It  applies  to  a  cohort  of  applicants  for 
accession  to  the  Air  Force.  Each  of  n  applicants  can  be  selected  for  one  of  M 
AFSC’s  or  can  be  rejected.  Table  1  summarizes  the  definitions  of  the  symbols  used 
to  express  this  framework. 

Setting  the  Goal;  Minimum  Cost .  This  framework  assumes  that  the  Air 
Force  wants  to  minimize  the  discounted  total  first  term  costs  — C* — of  meeting 
productivity  capacity  requirements  for  each  AFSC  with  specific  confidence  levels. 
Total  first  term  costs  is  defined  as: 

m  n 

C*  =  £  £  CyXij  (2) 

1=1  J=\ 


Here  Cy  stands  for  the  total  discounted  first-term  costs  expected  if  applicant  j  is 
selected  and  assigned  to  AFSC  i.  It  includes  the  costs  of  recruiting,  training, 
transporting,  and  maintaining  applicant  j  if  he  is  selected  and  assigned  to  AFSC  I 
and  remains  on  active  duty.  The  total  cost  for  each  month  is  multiplied  by  the 
probability  that  applicant  j  will  still  be  on  active  duty  during  that  month.  While  the 
task  of  bringing  together  the  data  to  calculate  the  Cy’s  might  require  substantial 
effort,  it  does  not  pose  significant  conceptual  or  data-availability  problems. 

The  variable  xy  is  restricted  to  the  values  zero  and  one.  If  applicant  j  is 
selected  and  assigned  to  AFSC  i,  then  Xy  =  1.  Otherwise,  Xy  =  0.  From  a 
mathematical  perspective,  the  problem  is  to  identify  the  sets  of  xy  =  1  that  minimizes 
Eq.  2  while  ensuring  that  the  constraints  discussed  below  are  met. 

Identifying  the  Constraints.  The  framework  assumes  that  the  Air  Force  needs  to 
obtain  Ri  of  total  productive  capacity  from  the  cohort  of  each  AFSC  i.  It  also 
assumes  that,  recognizing  the  random  error  inherent  in  forecasting  the  first-term 
productivities,  the  Air  Force  wants  the  probabilities  of  not  obtaining  the  Ri’s  be  no 
greater  than  some  set  of  pi’s  (0  <  pi  <  1).  Drawing  on  stochastic  programming 
methods,  (Taha,  1982,  pp.  784-788),  the  m  constrains  can  be  written  in  the  form: 
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Table  1 


Symbols  Used  in  Mathematical  Expressions 


Symbol 

a 


b' 


C* 

Cij 


Hi 


Ri 


Definition 

The  expected  increase  in  journeyman-equivalent  man- 
years  during  the  first  term  in  specialty  I  associated  with 
one  unit  increase  in  the  on  the  test  of  evaluation  criteria,  k 
=  0,1,..., K 

The  values  at  the  1  breakpoints  of  the  approximation 
internal  for  the  specialty  1  =  04^.  They  are  required  for  the 
separable  programming  solution  method  used  in  this 
paper. 

The  total  first-term  cost  of  satisfying  journeyman- 
equivalent  man-year  requirements  Ri,R2, ...» Ri,  ...Rm  with 
indicated  confident  levels 

The  expected  first-term  cost  of  assigning  applicant  j  to 
specialty  i.  It  includes  the  costs  of  recruitment,  training, 
and  transportation  as  well  as  the  costs  of  pay  and 
allowances.  Each  month’s  costs  in  multiplied  by  the 
probability  that  airman  j  will  still  be  on  active  duty  in  that 
month  if  assigned  to  specialty  i. 

The  matrix  of  test  scores  used  to  estimate  the  journeyman- 
equivalent  man-years  equation  for  specialty  i. 

The  score  of  the  jth  applicant  on  the  kth  test  or  factor  used 
to  predict  journeyman-equivalent  man-years  for  specialty 
i.  k=l,...,K. 

The  vector  of  scores  for  applicant  j  expressed  as  the 
difference  from  the  mean  of  the  scores  used  to  estimate  the 
equation  for  specialty  i. 

Expected  first-term  productive  capacity:  the  journeyman- 
equivalent  man-years  during  the  first  term  if  applicant  j  is 
assigned  to  specialty  i. 

The  number  of  journeyman-equivalent  man-years 
required  in  specialty  I  from  members  of  the  cohort. 

The  probability  that  applicant  j  will  be  on  active  duty  at  t 
months  if  assigned  to  AFSC  i. 
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s* 


s* 


tp 


VAR  (my) 


W* 


y(t)ij 


z" 


ZP 
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z 


w 


The  slack  variable  associated  with  applicant  j.  If  applicant 
j  is  selected  to  enter  the  Air  Force,  =  0.  If  applicant  j  is 
not  accepted,  S“  =  1. 

The  surplus  variable  associated  with  the  man-years 
requirement  for  the  ith  specialty. 

The  common  variance  of  the  linear  regression  equation 
used  to  predict  journeyman-equivalent  man-years  for 
specialty  i. 

The  standard  normal  value  required  to  ensure  that  the 
journeyman-equivalent  man-year  requirement(Ri)  is  met 
or  exceeded  with  a  probability  of  1-p  in  specialty  i. 

The  variance  of  the  jth  applicant’s  journeyman-equivalent 
man-years  during  his  first  term. 

The  non-negative  weights  associated  with  the  1  breakpoints 
of  the  approximation  interval  for  specialty  i.  No  more  than 
two  adjoining  weights  can  be  positive.  They  are  required 
for  a  separable  programming  solution  method.  1  =  0,...,L. 

Indicates  whether  applicant  j  is  assigned  to  specialty  1.  If 
applicant  j  is  assigned  to  specialty  i ,  the  xy  =  1. 

Otherwise,  Xy  =  0. 

The  expected  productivity  of  applicant  j  in  AFSC  i  at  t 
months  on  the  job  as  a  fraction  of  the  average  productivity 
of  journeymen  airmen  with  four  years  of  service. 

The  change  in  total  first  term  cost  if  a  second  applicant 
identical  to  applicant  j  was  available  to  enlist;  the  “shadow 
price”  of  applicant  j. 

The  change  in  the  total  first-term  costs  associated  with  an 
increase  in  the  sum  of  the  variance  for  the  ith  specialty  by 
one  journeyman-equivalent  man-year. 

The  change  in  the  total  first-term  cost  associated  with 
increasing  the  specialty’s  requirement  (Rj)  by  one 
journeyman  equivalent  man-year.  The  “shadow  cost”  of 
the  ith  specialty’s  requirement 

The  change  in  total  first-term  costs  associated  with  the  ith 
specialty’s  staffing  safety  margin. 
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Assinning  that  each  mij  is  normally  distributed  and  defining 


di  =  m  ijx ij 

7=1 


Then,  dj  is  normally  distributed  with  : 


E{di]-'^E{inij\Kij 


and 

VAl{d.]M'^DM 


where 

M  j-  =  ,1  >•••>  ^  in  ^ 

^  VAR{mi\)  .  COV {mn,  m^n) 

•  • 

Di  — 

,  COF(m.„,  Wii)  .  VAR  (min) 
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Now, 


R  -  Bid) 

-  -  ^ 

4VAIid) 


>\-P, 


Where 


is  the  standard  normal  variable  with  mean  zero  and  variance  one.  This  means: 


P  i{di  >  Ri}=  O 


Ri-  E  {di) 

^  ^VAR  {di)  j 


where  <I>  is  the  CDF  of  the  standard  normal  function.  Let  t^’i  be  the  standard 
normal  variable  such  that: 

Then,  the  statement 

>R 
i  i 

Is  realized  if  and  only  if 

R  i  -  E  i{d  i)  ^  ^  p 

This  yields  the  following  nonlinear  constraints: 

YE{m,j  \c,j  -  tf  4M^DiM  >  R,  (5) 

J  =  1 

for  1  =  l,...,m.  These  nonlinear  constraints  are  equivalent  to  the  original  stochastic. 
Since  the  distributions  of  the  my  are  independent. 
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COV(mij,...,mij)  =  0 


And  Eq.  (5)  can  be  written, 

-  It  Jt  (”  »  ^  («) 

7=1  V  > 

For  i=  1,...,  m- 

Eq.  6  makes  clear  an  important  implication  of  the  framework.  When  person 
j  is  assigned  to  AESC  i,  he  will  contribute  both  positively  and  negatively  towards 
satisfying  the  total  product  capacity  requirement  for  his  AFSC.  His  positive 
contribution  is  represented  by  the  expression  E(mij)— his  expected  equivalent  man- 
years.  The  negative  aspect  is  represented  by  his 

VAR{^y\l 

portion  of  the  second  term  on  the  left-hand  side  of  Eq  6.  The  second  term  stands  for 
the  staffing  safety  margin  necessary  to  ensure  the  Rj  is  met  or  exceeded  1  -  pi 
percent  of  the  time.  Selecting  and  assigning  individuals  whose  productivity 
capacities  can  be  forecast  only  with  large  errors  is  more  costly  because  the  required 
staffing  safety  margins  must  be  larger.  A  smaller  VAR(my)  is  as  much  a  cost- 
reducing  attribute  as  is  a  larger  my. 

One  can  solve  a  programming  problem  with  nonlinear  constrains  like  Eq.  6 
with  separable  programming  methods.  Following  Tana  (1982,  pp.  768-774),  one  can 
put  the  constraints  in  separable  programming  form  with  the  substitution. 


)  ^  ,y 


So,  each  of  the  original  constraints  are  equivalent  to  the  following 


2  E(my)Xy  -  tfq,  >  R, 
y  =  i 


(7a) 


and 


Y,  VAR  (m  ^  )x‘  -  9,^  =  0 

J  =  I 


for  i  =  m.  Since  Xij  =  0,1?  one  can  write  the  second  constraint: 

f^VAR  (m,)x,-qf  =  0  (7b) 

y=i 

One  can  eliminate  the  nonlinearities  arising  from  the  q^i  terms  by  dividing  the 
possible  ranges  for  the  q’s  into  1  segments.  Let  b'l  stand  for  the  values  of  qi  at  the 
limiting  values  of  its  range  and  at  its  breakpoints.  Then, 


and 


1=0 

where  the  wVs  are  weights.  The  larger  the  number  of  breakeven  points,  the  better 
the  approximation.  No  more  than  two  adjoining  wVs  can  be  positive  and  their  sum 
must  equal  one,  e.g.; 

2  ^ 1  ) 

1  =  0 

for  i  =  1, . . . .,  m. 

Substituting  for  qi  and  q^i  in  Eqs.  7a  and  7b,  respectively, 

2  -  tj’Y,  b‘w‘  >  R,  (9a) 

7=1  /=0 
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and, 


yVAR(m,)x,-t{b:)w;=0  (9b) 

y  =  1  /  =  o- 

for  i=  1, . . . . ,  m. 

Another  set  of  constraints  is  necessary  to  ensure  that  applicant  i  is  not 
assigned  to  more  than  one  position, 


(10) 

i=i 


Assembling  the  Analytic  Framework.  We  are  not  ready  to  assemble  the 
analytic  framework.  It  involves  minimizing  the  expression  at  Eq.  2  subject  to  the 
expressions  at  Eqs.  9a,  9b,  8,  and  10.  The  framework  can  be  restated  in  standard 
form  as  follows: 


m  n  ffi  L  m  n 

Mine  •  =  II  QXy  +  X I  (ok'  +  Z  (O^  +  Z  (O^  (2’) 

/=]  j=\  (=1  /=o  1=1  y=i 


subject  to: 

I  E{m,j  Z  ~  ^ 

y=i  /=0 


for  i  =  1, . . . . ,  m. 

I  VAR  (m,j )x^  -  I  (6/ )  w'  =  0  (9b) 

7=1  /=0 


for  i  =  1, . . . . ,  m. 

^  w'  =  1  (8) 

/  =  0 
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for  i  =  1, . . . . ,  m. 

m 

Y,x,j+S^j=\  (10-) 

»  =  l 

for  j^*!^****^  n*  ) 

;c,  =  0,1; w/,5/,5;  >  0 

Here,  the  S®i’s  are  the  surplus  variables  for  the  AFSC  required  productive  capacity 
constraints  and  the  S*j’s  are  the  slack  variables  in  the  applicant  constraints.  A 
particular  S®i  is  positive  if  the  cost  minimizing  selection  and  assignment  of 
applicants  involves  assigning  more  productive  capacity  to  the  ith  AFSC  than  is 
required  to  meet  Ri  with  the  required  confidence  level.  A  particular  S“j  =  1  if 
applicant  j  is  not  selected  for  Air  Force  service. 

Solving  the  Problem.  The  above  framework  represents  a  zero-one  mixed 
integer-programming  problem.  Taha  (1975,  pp.  113-119, 154-162,  and  346-347)  has 
identified  and  described  four  possible  algorithms  for  solving  zero-one  mixed  integer 
programs. 

However,  an  off-the-shelf  computerized  algorithm  might  not  be  available  for 
solving  the  above  problem.  The  restriction  on  the  w'j  weights  mentioned  above 
reduce  the  likelihood  that  such  an  off-the-shelf  algorithm  exists.  Recall  that  for  a 
particular  AFSC  i,  no  more  than  two  w'i’s  can  be  positive  and  that  the  two  positive 
w  i’s  must  be  adjacent.  This  restriction  means  that  the  corresponding  continuous 
linear  programming  problem,  which  usually  must  be  solved  to  provide  a  starting 
point  for  the  zero-one  mixed  integer  algorithm,  must  be  solved  under  the  conditions 
of  restricted  bases  (Taha,1975,  pp.  771-774).  The  implications  of  restricted  bases  for 
solving  the  zero-one  mixed  integer  problem  does  not  seem  to  have  been  fully 
explored  in  the  operations  research  literature. 
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IV.  The  Benefits  and  Better  Testing  and  Training  Methods 

The  analytic  framework  constructed  above  provides  ways  of  estimating  the 
benefits  to  the  Air  Force  of  improvements  in  assessment  methods  and  in  training 
programs.  It  provides  a  single  context  in  which  the  benefits  of  alternative 
improvements  in  assessment  methods  and  training  programs  can  be  compared. 

The  Benefits  of  Improving  Assessment  Methods.  Better  assessment  methods 
predict  job  performance  more  accurately.  The  measure  of  job  performance  used  in 
this  paper  in  journeyman-equivalent  man-years  (my).  The  simplest  improvement  in 
an  assessment  method  is  one  limited  to  a  single  AFSC.  Within  the  analytic 
framework,  this  kind  of  improvement  would  translate  into  reductions  in  the 

VAR(my)’s  for  the  ith  AFSC.  So,  the  IVAR(my)Xij  term  in  the  Eq.  9b  for  AFSC  i 
above  would  fall.  To  maintain  the  equality  of  this  Eq.  9b,  the  two  positive  adjacent 
w*!  weights  must  shift  “downward  towards  (bV ’s  with  lower  values.  As  a  result,  the 

t'‘iEb'iw'i  term  in  the  Eq.  9a’  for  AFSC  i  also  must  decline  in  value. 

If  the  Eq.  9a’  for  the  ith  AFSC  is  binding  (i.e.,  S*i  =  0),  then  the  term 
2VAR(my)xy  must  fall.  The  Air  Force  will  be  able  to  satisfy  at  a  1-pi  level  of 
confidence  the  total  equivalent  man-years  requirement — Ri — by  assigning  fewer 
applicants  to  the  ith  AFSC.  Since  fewer  applicants  are  required,  the  value  of  the 
objective  function— C*— will  decline.  The  approximate  benefit  of  the  assessment 
method  improvement  is  the  difference  between  the  values  of  C*  with  the  original 
assessment  method  and  with  the  improved  one. 

If  the  constraint  for  AFSC  i  represented  by  Eq.9a’  is  not  binding  (i.e.,  S*i  >0), 

f  fall  in  the  E VAR(my)Xij  term  might  still  lead  to  a  decline  in  C  .  A  decline  would 
occur  if  the  zero/one  nature  of  the  xy’s  made  it  necessary  to  assign  more  than 
enough  applicants  to  the  ith  AFSC  in  order  to  equal  or  exceed  Rj.  If  the 
EVAR(my)Xij  term  falls  enough  to  allow  one  or  more  “whole”  applicants  to  be  saved, 
then  C*  could  decline.  However,  a  decline  would  not  result  if  Eq.  9a’s  fails  to  be 
binding  for  some  other  reason. 

This  analytic  framework  can  deal  with  more  complex  improvements  in 
assessment  methods.^  It  can  deal  with  improvements  for  a  group  of  AFSCs  or  for 
all  AFSC’s.  In  all  cases,  the  benefit  is  the  difference  in  the  values  of  the  objective 
function —  C* — before  and  after  the  improvement. 

The  Benefits  of  Improving  Training  Programs.  In  terms  of  this  analytic 
framework,  a  new  training  program  could  have  impacts  via  three  routs. 
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•  A  new  training  program  would  alter  the  Cy  coefficients  in  Eg.  2  for  the 
AFSC’s  served  by  the  training  program.  A  lower  cost  training  program 
would  reduce  the  Qj’s  and  a  higher  cost  one  would  increase  them. 

•  A  new  training  program  would  impact  the  expected  journeyman- 
equivalent  man-years  term  — the  E  (mij)’s — in  Eqs  9a’  for  the  impacted 
AFSC’s.  An  increase  in  the  E  (mij)’s  would  mean  that  fewer  applicants 
would  have  to  be  to  the  AFSC’s  to  ensure  that  their  total  equivalent  man- 
year  requirements  (their  Ri’s)  were  achieved  with  the  desired  level  of 
confidences.  For  example,  Nichols,  Pokorny,  Jones,  Gott,  and  Alley 
(1989)  have  shown  that  an  intelligent  tutoring  system  can  substantially 
increase  trouble-shooting  efficiency  in  an  AFSC.  Such  an  increase  would 
represent  an  increase  in  expected  journeyman-equivalent  man-years. 

•  The  outcomes  of  different  training  programs  can  have  different 
variances  (Bloom,  1984).  The  impact  of  a  new  training  program  with  a 
lower  variance  would  follow  a  path  similar  to  the  one  described  above  for 
an  improved  assessment  method. 

A  new  training  program  that  lowered  the  Cy’s,  the  increased  the  E(mij)’s,  and  that 
reduced  the  VAR(mij}  could  clearly  provide  cost  avoidance  benefits.  Just  as  clearly, 
one  that  increased  the  Cy’s,  the  reduced  the  E(mjj),  and  that  increased  the 
VAR(mij)’s  would  be  wasteful.  Whether  other  new  programs  would  produce  new 
positive  benefits  would  depend  on  the  directions  and  relative  magnitudes  of  their 
three  impacts. 
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V.  Useful  Information  from  the  Dual  Problem 

Linear  programming  problems  have  associated  dual  problems,  which  can 
provide  additional  useful  information.  In  this  paper,  the  main  or  “primal”  problem 
involves  minimizing  costs.  If  a  primal  a  primal  problem  minimizes  something,  it 
dual  problem  wOl  maximize  something  else;  moreover,  primal  problem’s  minimum 
will  equal  its  dual  problem’s  maximum.  Following  Taha  (1982,  pp.  106-110),  the 
dual  problem  can  be  written  as  follows: 


Til  Tit  Til  Tl 

MaxZ'  =  'ZR,z;  +£(o)Z,'’  +IZ,"'  +IZ“  (11) 

,=1  j=l  1=1  J=1 

subject  to: 

£(m, )z;  +  VAR  (m, )z,'’  +  z;  <  C,  (12) 

for  all  combinations  of  i  and  j,  and  subject  to: 

-t^biz:  +  {b‘yzf  +  zr  <0  (13) 

for  each  1  within  each  i,  and  subject  to  : 

Z,"  >  0;Z’’  ,Z”unrestrict  ed;Zj  <  0 


The  primal  problem  represents  an  attempt  to  minimize  costs  subject  to 
constrains  imposed  by  staffing  and  staffing  safety  margin  requirements  for  each 
specialty,  by  our  capability  to  predict  on  the  job  performance  in  each  specialty,  and 
by  characteristics  of  the  individuals  in  the  applicant  pool.  In  the  duel  problem,  Z 
represents  that  sum  of  the  implied  values  of  the  specialty  staffing  and  staffing  safety 
margin  requirements,  of  the  capability  to  predict  one  the  job  performance  in  each 
specialty,  and  of  particular  members  of  the  applicant  pool.  The  dual  problem 
maximizes  the  implied  values  of  the  constraints.  In  other  words,  the  solution  that 
minimizes  the  costs  of  meeting  the  constraints  maximizes  the  marginal  values  of 
“loosing”  the  constraints  that  have  proven  to  be  binding. 

The  other  four  Z  variables  are  the  implied  or  “shadow”  marginal  cost  of 
meeting  each  of  the  constrains  in  the  primal  problem.  There  is  a  Z  variable  for 
each  constraint  in  the  primal  problem;  the  these  four  Z  variables  correspond  to  the 
four  kinds  of  constraints  equations  in  the  primal  problem.  The  can  be  interpreted 
as  follows: 
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Z''i  the  change  in  the  total  first-term  costs  that  would  result  from 
increasing  the  ith  speciality’s  staffing  requirement  (Ri)  by  one 
journeyman-equivalent  many-year.  It  represents  the  marginal 
shadow  cost  of  the  ith  speciality’s  man-year  requirement. 

Z^i  the  change  in  the  total  first-term  costs  that  would  result  form 

increasing  the  sum  of  the  variances  in  predicted  performance  in  the 
ith  specialty  by  one  journeyman-equivalent  man-year.  The  negative 
of  this  value  provides  an  estimate  of  the  incremental  value  of 
predicting  on-the-job  performance  in  the  ith  specialty  more 
accurately. 

T^i  the  change  in  total  first-term  costs  that  would  result  from  increasing 
the  ith  speciality’s  staffing  safety  margin. 

Z^i  the  change  in  total  first  term  cost  that  would  result  if  another 

applicant  identical  to  applicant  j  in  all  ways  was  available  to  enlist.  It 
represents  the  incremental  value  of  adding  another  applicant  like 
applicant  j  to  the  applicant  pool. 

The  presents  of  the  Xy  integer  variables  in  the  primal  problem  can  cause  the 
values  of  some  of  the  shadow  cost  variables  to  be  misleading  (Baumol,  1965,  pp.l63- 
166).  The  shadow  cost  variables  corresponding  to  primal  problem  constraints  that 
contain  integer  variables  could  provide  misleading  information  about  costs. 
Procedures  have  been  developed  to  recalculate  such  shadow  costs  variables  so  that 
they  reflect  better  economic  costs  (Alcaldy  &  Klevorick,  1966).  However,  they 
would  not  be  needed  for  our  analysis,  since  it  uses  the  shadow  costs  and  the  Z^’^j’s 
Correspond  to  constraints  in  the  primal  problem  that  contain  only  continuous  w'l 
variables. 

Above  we  interpreted  the  Z"  shadow  costs  as  the  changes  in  the  total  first- 
term  costs  associated  with  specialty  i’s  staffing  safety  margin.  This  interpretation 
needs  a  better  explanation.  Please  review  Eq.8,  which  corresponds  to  a  particular 
Z*  .  For  the  moment,  assume  that  it  equals  zero.  But,  if  all  the  w'i  weights  for  the 
ith  AFSC  were  zero,  then  the  Eq.  9b  corresponding  to  the  ith  AFSC  would  require 

2)VAR(mij)xij  =  0,  and.  So,  the  staffing  safety  margin  term  in  Eq9a’  corresponding 
to  the  ith  AFSC  would  disappear.  As  a  result,  the  total  first  term  cost  of  satisfying 
the  journeyman-equivalent  man-year  requirement  (C‘)  would  fall  since  the  Air 
Force  no  longer  would  have  to  select  and  assign  additional  applicants  to  satisfy  the 
ith  speciality’s  staffing  safety.  So,  the  Z”  variable  must  reflect  the  cost  of  providing 
the  staffing  safety  margin  for  the  ith  AFSC. 

In  the  last  section,  we  saw  that  an  improvement  in  assessment  methods  for 
am  AFSC  led  to  a  reduction  in  its  required  staffing  safety  margin.  Let  7^\ 
represent  the  shadow  costs  of  the  required  staffing  safety  margin  with  the  old 
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assessment  methods  and  iT-l  represent  the  shadow  cost  of  the  required  staffing 
safety  margin  with  the  new  assessment  methods.  Then,  the  cost-saving  beneflt 
resulting  from  the  improvement  in  assessment  methods  for  the  ith  AFSC  is  given  by 

q=^-2r'  (14 

In  the  last  section,  we  say  that  the  difference  in  the  values  of  the  primal  problem’s 
objective  function  before  and  after  an  improvement  in  assessment  methods  or  a 
training  program  for  the  ith  AFSC  represented  the  total  cost-saving  benefit.  Eq  14 
gives  the  cost-saving  benefit  associated  with  a  reduction  in  the  required  staffing 
safety  margin.  It  is  a  component  of  total  benefit.  The  difference  between  the  two 
benefits  can  be  viewed  as  the  cost-saving  benefit  due  to  improved  selection  and 
greater  exploitation  of  the  individuals’  comparative  advantages. 
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VI.  Concluding  Comments 


The  framework  put  forward  in  this  paper  has  several  advantages: 

•  It  represents  the  only  framework  that  includes  all  three  sources  of 
benefits — the  selection  of  more  productive  applicants,  the  exploitation  of 
their  comparative  advantages  in  different  jobs,  and  the  reduction  of 
staffing  safety  margins. 

•  By  using  the  journeyman-equivalent  man-year  concept  and  structuring 
the  problem  as  a  cost  minimizing  one,  it  avoids  the  industrial 
psychologist’s  problem  of  having  to  define  and  quantify  “utility” 

•  The  framework  can  deal  with  the  benefits  of  both  improvements  in 
assessment  methods  and  improvements  in  training  programs;  it  is  the 
only  framework  that  puts  improvements  assessment  methods  and 
training  programs  on  the  same  footing  and  allows  their  benefits  to  be 
compared. 

•  It  builds  on  demonstrated  mathematical  and  statistical  methods. 


These  advantages  have  a  price.  The  analytic  framework  makes  large 
demands  of  quantitative  inputs;  using  the  framework  to  analyze  meaningful 
problems  would  require  major  data  collection  and  analysis  efforts.  Also,  the 
analytic  framework  draws  on  advanced  operation  research  techniques — stochastic 
programming,  separable  programming,  and  mixed  integer  programming.  Their 
application  to  an  actual  problem  is  likely  to  involve  much  more  than  running  an  off- 
the-shelf  computer  program  (Taha  1975,pp.  346-47).  Existing  algorithms  might 
have  to  be  modified  or  new  ones  might  have  to  be  written. 

Viewed  from  another  perspective,  an  important  function  of  a  analytic 
paradigm  is  to  clarify  concepts  and  to  identify  research  problems.  Moreover,  a  set 
of  researchable  problems  can  represent  a  research  agenda.  Hopefully,  the  analytic 
framework  described  in  this  paper  has  been  useful  in  doing  both. 
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Notes 


1.  An  interesting  special  case  would  arise  if  journeyman-equivalent  man-years 

could  be  predicted  with  a  linear  regression  equation  like 

niy  =  +  a]  hi  + . + 

where  h^  (k=l,.-»K)  stands  for  the  score  of  the  jth  canadate  on  the  kth  assessment 
testof  method  and  aV...,aNare  estimated  regression  coefficents.  Substituting  the 
predicted  journeyman-equivalent  man-years  of  canadate  j  if  selected  andassigned  to 
AFSC  i  for  E(mij)  and  the  standard  errors  of  prediction  (Intriligator,  p.  112)  for  the 
variances  in  the  ith  AFSC  in  Eqs  9a  and  9b,  respectively, 

7=1  /=0 

= 0  (96) 

7=1  /=o 

where 

S^i  is  the  common  variance  of  the  linear  regressionequation  used  to  predict 
joumey-man  equivalane  man-years  for  AFSC  i. 

hij  is  the  vector  of  scores  fo  rcanadate  j  expressed  as  differences  from  the  mean 
of  the  populationused  to  estiamte  the  regression  equation  for  AFSC  i. 

Hi  is  the  matric  of  scores  used  to  estiamte  the  regression  equation  for  AFSC  i. 

Note  that  the  standard  error  of  prediction  will  be  larger  for  applicants  with 
very  high  and  very  low  assessment  scores.  In  this  special  case,  assigning  an 
applicant  with  extreme  assessment  scores  is  inherently  riskier  than  assigning  an 
applicant  with  “average”  assessment  scores.  If  regression  equations  were  used  to 
predict  equivalent  man-years,  applicants  with  very  low  assessment  scores  would  be 
burdened  in  the  competitionfo  rAir  Force  jobs  not  only  by  eheir  low  predicted 
perforamnce  but  also  by  the  greater  risk  associated  with  realizing  their  predictions. 
At  the  other  extreme,  canadates  with  very  high  assessment  scores  would  experience 
some  “drag”  on  their  predictions  of  superior  job  performance  due  to  the  larger 
risks  associated  with  realizing  their  predictions. 
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